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Its Really About Model Credibility*. NASA 

Achieving a high level of belief or trust in the model 


• NASA-STD-7009 

- Standard for Models and 
Simulations (M&S) 

• Eight Factors 

- Verification 

- Validation 

- [nput Pedigree 

- Results Uncertainty 

- Results Robustness 

- Use History 

- M&S Management 

- People Qualifications 


Credibility Assessment 
Factors 

Evidence 

Technical Review 

Factor 

Score 

Weighted 
Subfactor Score 

Overall 

Score 

Sufficiency 

Threshold 

Score* 

Weight* 

Threshold* 

Score* 

Threshold* 

1 

Verification 

2 

0.20 

3 

2 

3 

2 

0.40 

1.75 

2.54 

2 

Validation 

2 

0.25 

2 

2 

3 

2 

0.50 

3 

Input Pedigree 

2 

0.10 

3 

2 

3 

2 

0.20 

4 

Results Uncertainty 

0 

0.10 

2 

0 

3 


0.00 

5 

Results Robustness 

2 

0.10 

2 

2 

3 

2 

0.20 

6 

Use History 

1 

0.15 

2 

N/A 

N/A 

1 

0.15 

7 

M&S Management 

2 

0.05 

3 

N/A 

N/A 

2 

0.10 

8 

People Qualifications 

4 

0.05 

3 

N/A 

N/A 

4 

0.20 


* Maximum = 4; where 0=insufficient evidence and 4=highest fidelity/rigor achievable 
+ Minimum = 0.05, maximum = 0.25 and sum of all weights must equal 1.0 


Generic STD-7009 Credibility Assessment 

Verification 



Robustness 
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Problem with “Input Pedigree” Ranking 


• A complex model integrating many factors can have a wide 
array of input pedigree scores across the range of model 
parameters 

- Source, quality of the data, type of data 

• From NASA-STD-7009: Input Pedigree (Ranking) involves 
the evaluation of all data that is used as input for the current 
M&S results. It includes not only data that is unique to the 
model, but also data that is produced by other simulations. 

• Implication: Model and Simulation Input Pedigree Ranking is 
equivalent to the lowest input pedigree of the model 
parameters (i.e. logical AND). 
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Question: 
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* In practice, the Input Pedigree ranking of complex multi- 
parameter models tends to be low, resulting in lower than 
expected model and simulation credibility. 

* Question: Is this evaluation of credibility true for all 
applications and simulations of the model? 


* Considerations: 

- Application credibility differs from the model credibility with respect to 
input pedigree. 

- Application credibility can be more precisely determined using 
Sensitivity analysis (also known as Results Robustness) to address 
relative importance of input parameters. 
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Sensitivity Analysis Methodology 

• 


• Saltelli: “Sensitivity Analysis is the study of how variation in the 
output of a model can be apportioned, qualitatively or 
quantitatively , to different sources of variation (input) and how the 
given model depends upon the information fed into it.” 

• Partial Rank Correlation Coefficient (PRCC) Analysis 

- Provides the linear relationships between two variables (one input parameter 
and one output parameter) when all linear effects of other variables are 
removed after rank transformation. 

- Rank Transformation: transforms non-linear monotonic relations to linear. 

• KEEP IN MIND the difference in a contributing parameter and a 
sensitive parameter 

- Many parameters contribute substantially to the mean output of the model 

• Low sensitivity may indicate a “DC-signal effect” over the range of model application 
and parameter variance 

- Parameter variance affecting model output (magnitude and variance) indicates 
a sensitive parameter 
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Using IMM for ISS mission 6 crew members 


• IMM Provides probabilistic analysis of 99 medical condition 
occurrences and impact to mission outcomes 

• Context : 6-month, 6 crew (M-4, F-2), ISS med kit, and evacuation is 
possible 

• Output: 

- Crew Health Index (crew available time - time lost due to medical events) 

- Probability of consideration of evacuation 




Sensitivity Analysis Results: Crew Health Index for 
6 Month Mission to the International Space Station 


VISUAL IMPAIRMENT — 
SKIN LACERATION - 
DENTAL CARIES- 
DENTAL ABSCESS- 
EYE CHEMICAL BURN- 
HEARING LOSS - 
NEPHROLITHIASIS - 
SKIN ABRASION - 
LOWER EXTREMTY STRESS FRACTURE - 
STROKE CEREBROVASCULAR - 
DENTAL EXPOSED PULP - 
EYE INFECTION - 
TRAUMATIC HYPOVOLEMIC SHOCK - 
SEPSIS - 

BAROTRAUMA EAR SINUS BLOCK - 
HIP PROXIMAL FEMUR FRACTURE - 
URINARYTRACT INFECTION - 
SMALL BOWEL OBSTRUCTION - 
EYE ABRASION FOREIGNBODY — 
APPENDICITIS - 
ANGINA MYOCARDIAL INFARCTION - 
BACK INJURY — 
WRISTFRACTURE - 
FINGERDISLOCATION — 
ATRIAL FIBRILLATION - 

- 0.5 



- 0.45 


- 0.4 


- 0.35 


- 0.3 


- 0.25 


- 0.2 


- 0.15 


- 0.1 


- 0.05 
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Sensitivity Analysis Results: Consideration of Evacuation for 6 
Month Mission to the International Space Station 


NEPHROLITHIASIS 
STROKE CEREBROVASCULAR 
DENTAL ABSCESS 
TRAUMATIC HYPOVOLEMIC SHOCK 
HIP PROXIMAL FEMUR FRACTURE 
SEPSIS 

SMALL BOWEL OBSTRUCTION 
ATRIAL FIBRILLATION 
ANGINA MYOCARDIAL INFARCTION 
SEIZURES 
HEAD INJURY 
RETINAL DETACHMENT 
WRIST FRACTURE 
APPENDICITIS 
ACUTE COMPARTMENT SYNDROME 
ACUTE DIVERTICULITIS 
SHOULDER DISLOCATION 
CHEST INJURY 
VISUAL IMPAIRMENT 
EYE PENETRATION FOREIGN BODY 
ABDOMINAL INJURY 
ACUTE CHOLECYSTITIS BILIARY COLIC 
SMOKE INHALATION 
ACUTE PANCREATITIS 
EYE CHEMICAL BURN 



0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 
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Interpretation: Weighted Averaging 

• 


• Weighted averaging of the sensitivity scores uses the 
PRCC output (W PRCC )to “weigh” parameter pedigree (IP|) 

• For this analysis assume 

- One input parameter ranked at 1 , the rest are 3 

- Originally a strict “Std-7009” Input Pedigree Score = 1 



Eye Chemical 
Burn - IP, otal 

100 

Conditions 

25 Most Influential 
Conditions 


CHI 

2.944 

2.918 

, _ I w PRCCii i Pi 

total y 1 T/i/ 

L v* PRCC, i 

Consideration 
of Evacuation 

2.772 

2.935 




VHP - IPtota. 

99 

Conditions 

25 Most Influential 
Conditions 


CHI 

2.963 

2.959 


Consideration of 
Evacuation 

2.691 

2.545 
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Interpretation: Threshold Specification 

• 


• Set a threshold value below which the contribution of the 
parameter is considered to be small 

- Pedigree of data below this level is not considered in the assessment of 
simulation input pedigree 

• Assume conservative threshold to be a PRCC =1/n = 0.01 and a 
more lenient threshold to be PRCC = 2/n = 0.02 

- n = the number of parameters 

• For this analysis assume 

- One input parameter ranked at 1 , the rest are 3 

- Strict “Std-7009” Input Pedigree Score = 1 
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Threshold Analysis Results 




• NASA 


Conservative 

Threshold 

+/- 0.01 


CHI 



Less 

Conservative 

Threshold 

+/- 0.02 


Consideration of Evacuation 
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Conclusions 
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• Proof of concept analysis indicates 

- Sensitivity analysis can be used to better characterize the input pedigree 
ranking of specific simulations 

- One analysis is insufficient to characterize the input pedigree ranking for all 
possible analyses - Case by case basis 

• Method used to evaluate input pedigree requires further 

investigation 

- Current sample approaches are reasonable but not optimized 

- Quantification should be repeatable and consistent with other communications 
components of 7009 to maintain decision making information quality 

• Future efforts will address 

- More appropriate definitions of cutoff thresholds, allowing a distinction in 
numerical noise and statistical significance 
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INTEGRATED MEDICAL MODEL 

OVERVIEW 


Thank you! 






CHI - All Conditions 


W 1 
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Consideration of Evacuation - All Conditions 


H Figures - 100k Runs ISS6 All PRCC ME 
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